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1.  Introduction 

Brier,  Zacks,  and  Marlow  (1985)  considered  the  problem  of 
simultaneously  estimating  a  set  of  vector  parameters  6^...  6^,  where  6^ 
is  a  vector  of  probabilities.  The  problem  was  motivated  by  the  Marine 
Corps  Combat  Readiness  Evaluation  System  (MCCRES),  which  is  designed  to 
measure  the  state  of  readiness  of  Marine  combat  units.  In  MCCRES  units 
are  tested  to  determine  whether  or  not  they  can  satisfactorily  perform 
a  number  of  requirements;  the  result  of  each  test  is  classified  as 
either  satisfactory  or  unsatisfactory.  The  requirements,  which  typically 
number  in  the  hundreds,  fall  into  one  of  k  categories  and  it  is  assumed 
that,  for  a  given  unit,  the  probability  of  satisfactorily  completing  a 
requirement  is  the  same  for  all  requirements  in  the  same  category.  In 
MCCRES,  the  categorization  of  requirements  is  defined  by  the  following 
six  categories: 
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1.  Advanced  preparation  and  education 

2.  Combat  service  support  and  logistics 

3.  Mission  planning  and  preparation 

4.  Command  control  and  task  organization 


5.  Execution 


6.  Information  and  communication 


The  available  data  consists  of  {N^.},  i=l,...n,  j=l,...,k,  the 

number  of  tested  requirements  in  the  jth  category  applicable  to  the  ith 

unit,  and  {X^.},  the  corresponding  number  of  requirements  that  were 

satisfied.  Brier,  Zacks,  and  Marlow  proposed  using  empirical  Bayes' 

estimators  of  {0 ^ }  as  an  improvement  over  the  sample  proportions 

(X../N..1.  The  Bayes'  estimators  are  based  on  the  arcsin  transformation 
1J  I/O 

of  the  proportions,  Y^.  =  2  arcsin  [  (X„  +  3/8)/ (N„  +  3/4)]  ,  and  the 

corresponding  transformation  of  the  probabilities,  r\. .  =  2  arcsin  /@ . . . 

The  assumptions  of  the  Bayes  model  are  that  r}.(r|.  =  (n . .  , . .  .y .,)  )  , 

il  ik. 

1=1,... n  are  independent  (y,£)  and  that,  conditional  on  rijj ,  X„  is 

binomially  distributed  and  therefore  Y„  is  distributed  approximately 

as  N(y..,  N.^).  These  assumptions  lead  to  Bayes'  estimators  of  the  form 

y.  =  Y.  -  A  (Y.  -  y),  where  A  is  a  function  of  £  and  {N  } . 

a,1  a.1  % 

Brier,  Zacks,  and  Marlow  (1985)  discussed  the  appropriateness  of 
the  above  assumptions.  They  obtained  empirical  Bayes  estimators  (EBE's) 

by  replacing  y  and  £  by  their  maximum  likelihood  estimates  which  were 

% 

computed  by  applying  the  EM  algorithm  of  Dempster,  Laird  and  Rubin  (1977). 
The  model  was  applied  to  the  data  of  Volume  II  of  MCCRES  and  the  EBE's 
were  compared  to  the  sample  proportions  as  estimators  of  the  true 
probabilities.  Cross-validation  techniques  clearly  demonstrated  that 
the  EBE's  provided  a  substantial  improvement  over  the  sample  proportions 
for  the  Volume  II  data. 

The  purpose  of  this  study  is  to  explore  the  performance  of  the 
EBE  across  a  wider  range  of  parameter  values  and  sample  sizes.  To 
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accomplished  this  we  used  simulation  techniques  as  described  below. 
Although  the  degree  of  improvement  obtained  by  the  EBE's  varies  some¬ 
what  over  the  values  of  the  parameters,  the  simulation  study  indicates 
that  the  EBE's  are  to  be  preferred,  in  general,  to  the  sample  proportions. 

2.  Results  of  the  Simulation  Study 

In  our  simulation  study  we  have  fixed  k  to  be  six,  corresponding 
to  the  MCCRES  categorization  of  requirements.  The  number  of  units,  n, 
was  either  50  or  100.  The  number  of  requirements  per  category,  , 
was  constant  over  all  (i,j),  i.e.,  =  N,  and  we  considered  three 

values  of  N  in  the  study,  N=5,25,50.  The  parameters  of  the  distribution 

of  ti  are  y  and  £,  the  mean  vector  and  covariance  matrix,  respectively. 

%  'v.  % 

We  set  all  six  components  of  y  equal  and  considered  y  =  1.57,  2.09,  and 

% 

2.50  in  the  study.  Note  that  these  means  are  in  the  arcsin  scale  and 

correspond  to  probabilities  of  .5,  .75,  and  .9,  respectively.  With 

2 

regard  to  I,  we  set  all  variance  equal  (o  )  and  all  correlations  equal 

2 

(p).  Our  study  fixed  o  at  either  .01  or  .10  and  p  at  either  0,  .4,  or 


Two  criteria  were  used  to  compare  the  performance  of  the  estimators. 
The  First  criteria  is  the  total  mean  square  error  of  estimation, 
n  6  2 

I  F  (rj..  -  y .  . )  .  These  measures  are  presented  in  Table  1. 
i=l  j=l  1J  1J 

Another  measure  of  performance  was  obtained  by  examining  the  absolute 
error  of  estimation  and  tabulating,  in  each  case,  the  proportion  of 
errors  that  are  greater  than,  .01,  .05,  .10,  and  .20,  respectively. 

Table  2  presents  the  results  of  this  comparison. 

In  addition  to  comparing  the  mean  square  errors  of  the  ordinary 
estimates  and  the  empirical  Bayes'  estimates,  we  also  considered  the 
mean  square  error  of  estimation  of  a  third  estimator. 
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where  Y.  =  (  E  N..  Y.  .)/(  E  N .  .  )  ,  and  a+  =  max  (0,a).  This  estimator 

j=l  3  3  j=l  3 

was  suggested  by  James  and  Stein  (1961)  and  was  considered  by  Brier, 

(  s  ) 

Zacks,  and  Marlow.  Table  1  demonstrates  that  ,  the  Stein  estimator, 

is  clearly  infer*->r  to  the  Bayes  estimator  and,  in  many  instances,  is 
inferior  to  the  sample  proportions.  These  findings  are  consistent  with 
those  of  Brier,  Zacks,  and  Marlow,  and  hence  the  Stein  estimator  will  not 
be  considered  further. 

An  examination  of  Tables  1  and  2  indicates  the  clear  superiority 
of  the  Bayes'  estimates  across  the  range  of  parameters.  Even  in  the  case 
where  p  =  0,  in  which  the  Bayes'  estimates  should  provide  the  least 

amount  of  improvement,  the  Bayes  estimates  are  almost  always  better  than 
the  ordinary  proportions,  especially  in  situations  where  the  sample  sizes 
are  small. 

A  number  of  important  aspects  of  the  results  should  be  noted. 
Although  the  performances  of  both  estimators  deteriorate  as  N  becomes 
small,  the  relative  advantage  of  the  Bayes'  estimators  increases  in  small 
samples.  This  is  to  be  expected  since  the  Bayes'  estimators  "borrow 
strength"  from  other  units  and  other  category  scores  within  the  same 
unit.  The  relative  performance  of  the  Bayes'  estimators  also  increases 
with  the  degree  of  correlation  between  category  scores:  when  p=.8, 
even  when  N=50  (and  the  individual  proportions  contain  an  appreciable 
amount  of_  information  about  the  corresponding  probabilities),  the  mean 
square  error  of  the  Bayes'  estimates  are,  at  worst,  of  the  order  of  one- 
half  that  of  the  ordinary  estimates.  But  even  when  p  =  0,  the  Bayes' 
estimates  are,  at  worst,  comparable  to  the  ordinary  estimates.  Finally, 
we  note  that  50  evaluations  appears  to  provide  a  more  than  adequate 
size  sample  to  estimate  y  and  E. 


3.  Conclusions 


The  results  of  this  study  strongly  support  the  conclusion  that 
empirical  Bayes*  estimators  will  provide  considerable  improvement  over 
the  sample  proportions  in  situations  where  the  category  probabilities 
are  positively  correlated  and  the  units  are  drawn  from  a  moderately 
homogeneous  population.  Although  we  have  only  studied  the  behavior  of 
the  Bayes'  estimators  for  k=6  categories,  we  would  expect  comparable 
results  for  larger  values  of  k  if  the  number  of  units,  n,  were  increased 
to  account  for  the  extra  parameters  in  y  and  £  that  would  require 
estimation. 
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TABLE  1 


Comparison  of  Mean  Square  Errors  for  Ordinary  Estimates 
Stein  Estimates,  and  Bayes'  Estimates 


Number  of 
Evaluations 


50 


100 


u  -  i 

.57 

Number  of 

o2=.l 

2 

O  -.01 

Observations 

Per  Category 

_ 

0 

p=.4 

£^8 

£=0 

£1:A 

p=.8 

Ordinary 

13.46 

12.62 

14.36 

15.48 

14.89 

14.20 

5 

Stein 

8.38 

5.48 

3.71 

2.64 

3.56 

2.34 

Bayes 

5.24 

4.87 

3.21 

1.25 

1.40 

1.25 

Ordinary 

2.71 

2.34 

2.53 

2.74 

2.70 

3.11 

25 

Stein 

4.40 

3.63 

1.73 

1.06 

0.82 

0.57 

Bayes 

2.09 

1.63 

1.40 

0.64 

0.66 

0.62 

Ordinary 

1.27 

1.30 

1.27 

1.57 

1.53 

1.61 

50 

Stein 

2.11 

2.10 

1.26 

0.78 

0.68 

0.35 

Bayes 

1.03 

1.01 

0.82 

0.56 

0.51 

0.41 

Ordinary 

27.92 

26.58 

26.10 

27.22 

29.88 

26.89 

5 

Stein 

15.89 

11.19 

7.03 

5.36 

6.86 

3.93 

Bayes 

9.71 

8.46 

6.46 

1.85 

2.44 

2.04 

Ordinary 

5.52 

5.53 

4.75 

5.93 

6.20 

5.58 

25 

Stein 

8.13 

5.95 

3.08 

1.99 

2.04 

1.18 

Bayes 

3.82 

3.50 

2.33 

1.36 

1.33 

1.06 

Ordinary 

2.77 

2.58 

2.77 

2.90 

2.81 

2.99 

50 

Stein 

4.60 

4. 63 

2.54 

1.81 

1.15 

0.79 

Bayes 

2.23 

2.07 

1.73 

1.16 

0.77 

0.85 
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Ordinary 

Stein 

Bayes 

Ordinary 

Stein 

Bayes 

Ordinary 

Stein 

Bayes 


19.52  21.99 

11.40  9.27 

7.23  6.43 

4.54  4.38 

8.04  5.29 

3.40  2.96 

2.20  1.95 

5.15  3.99 

1.90  1.54 


21.61  22.79  24.51  19.67 

5.43  3.98  4.24  3.19 

4.90  1.54  1.90  1.49 
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TABLE  1  -  (Cont'd) 


V  -  2. 

50 

Number  of 

a2=.l 

a2=.01 

Number  of 

Observations 

Evaluations 

Per  Category 

p=0 

p=.4 

£=0 

p=.4 

p=.8 

Ordinary 

5.00 

4.63 

5.48 

5.63 

6.05 

5.09 

5  Stein 

2.74 

1.99 

1.49 

1.12 

1.09 

1.28 

Bayes 

2.51 

2.42 

1.95 

0.68 

0.88 

0.73 

Ordinary 

1.04 

1.23 

0.97 

1.27 

1.08 

0.87 

50 

25  Stein 

2.17 

1.30 

0.65 

0.47 

0.30 

0.18 

Bayes 

0.79 

0.64 

0.58 

0.34 

0.27 

0.17 

Ordinary 

0.49 

0.58 

0.57 

0.52 

0.58 

0.64 

50  Stein 

1.77 

1.34 

0.54 

0.33 

0.25 

0.16 

Bayes 

0.43 

0.45 

0.30 

0.20 

0.22 

0.16 

Ordinary 

11.82 

10.03 

11.15 

10.88 

10.72 

9.70 

5  Stein 

5.87 

4.51 

2.88 

2.09 

2.14 

1.53 

Bayes 

5.10 

5.06 

3.51 

1.33 

1.29 

1.11 

Ord  inary 

2.34 

2.49 

2.36 

2.19 

2.11 

2.20 

100 

25  Stein 

1.69 

3.35 

1.46 

0.70 

0.62 

0.49 

Bayes 

4.04 

1.68 

1.21 

0.41 

0.41 

0.39 

Ordinary 

1.21 

1.09 

1.00 

0.97 

1.06 

1.00 

50  Stein 

3.65 

2.61 

1.09 

0.55 

0.46 

0.29 

Bayes 

1.03 

0.82 

0.61 

0.37 

0.33 

0.26 
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TABLE  2 

Distribution  of  Absolute  Errors,  for  Ordinary 
Estimates  and  for  Empirical  Bayes'  Estimates 


(a)  y  =  1.57,  o  -.10 


Number  of 
Evaluations 


Number  of 
Observations 
Per  Category  p 


Proportions  of  Errors  Greater  Than 


,98*  .96* 

,95  .94 

,98  .94 


.84*  .74* 

.77  .67 

.86  .63 

.58  .55 

.56  .47 

.55  .48 

.46  .39 

.46  .40 

.46  .33 


_ 

10 

.68* 

.46* 

.59 

.44 

.68 

.36 

.31 

.24 

.26 

.18 

.27 

.15 

.11 

.08 

.13 

.09 

.12 

.05 

.64 

.45 

.65 

.42 

.63 

.33 

.30 

.21 

.28 

.19 

.26 

.10 

.15 

.10 

.13 

.09 

First  entry  in  each  column  corresponds  to  the  ordinary  estimates 
second  entry  corresponds  to  the  Bayes*  estimates. 
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TABLE  2  -  (Cont'd) 


T-499 


T-499 


T-499 


>3 

I 

* 

II 

15 

i 

m 


TABLE  2  -  (Cont’d) 


(e)  y  =  2.50,  a  =.10 


Number  of 
Evaluations 

Number  of 
Observations 
Per  Category 

; 

p 

0 

5 

.4 

.8 

0 

50 

25 

.4 

.8 

0 

50 

.4 

.8 

1.00  .83 

1.00  .89 

.99  .90 

.90  .76 

.88  .72 

.85  .65 

.80  .70 

.82  .72 

.82  .68 


,94  .32 

,95  .41 

.93  .37 

,41  .16 

,38  .11 

.38  .04 

,25  .06 

.26  .06 
.26  .03 


.49  .02 

.51  .03 

,49  .01 

.12  .00 
.10  .00 
.06  0 

.01  0 
.02  0 
.03  0 


.96  .34 

.96  .33 

.94  .25 


.43  .05 

.40  .05 

.40  .05 

.20  .05 

.21  .02 
.21  .02 


